Accepting Failure: Availability through Repair-centric System Design

Motivated by the lack of rapid improvement in the availability of Internet server systems, we introduce a new philosophy for designing highly-available systems that better reflects the realities of the Internet service environment. Our approach, denoted repair-centric system design, is based on the belief that failures are inevitable in complex, human-administered systems, and thus we focus on detecting and repairing failures quickly and effectively, rather than just trying to avoid them. Our proposal is unique in that it tackles the human aspects of availability along with the traditional system aspects. We enumerate a set of design techniques for building repair-centric systems, outline a plan for implementing these techniques in an existing cluster email service application, and describe how we intend to quantitatively evaluate the availability gains achieved by repair-centric design.